io: backend-swappable I/O layer for ESP32-P4 (Tab5) port - #1
Open
yasunorioi wants to merge 7 commits into
Open
Conversation
Introduce src/io.zig wrapping std.net.Stream/Address behind io.Stream/ io.Address (surface: handle/read/writeAll/close, initIp4/parseIp/getPort/ format) plus io.tcpConnectToHost. All connection handlers now thread the io.* types instead of std.net.*; listen/accept stay on std.net.Server and convert at the accept boundary via fromNet. Adds build option -Dio-backend=posix|lwip (default posix). posix delegates 1:1 to std.net (host/Linux/cloud, behavior unchanged); lwip is an @CompileError seam marking io_lwip.zig TODOs (lwip_read/write/close, tcpConnectToHost) for the ESP-IDF/lwIP backend. Target: 5-10 concurrent connections keeps the thread-per-connection model, so no event-loop rewrite is needed; same source compiles to the cloud caster and the embedded firmware. zig build / test / test-integration all green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
io_lwip.zig: lwIP socket ops (read/writeAll/close via lwip_read/write/ close, tcpConnectToHost via lwip_getaddrinfo/socket/connect). io.zig Stream/tcpConnectToHost now dispatch to it under -Dio-backend=lwip (anyerror return types decouple signatures from std.net); posix path is byte-identical. io_lwip.zig only parsed on the lwip build (comptime-false import guard), so the host build never touches lwIP headers — NOT host- verifiable, it's the embedded seam. source.zig: extract SourceFeeder (chunk -> ring + telemetry + RTCM3 scan) shared by sourceLoop and the new runLocalSource. runLocalSource + ByteReader register a Source and feed it from an arbitrary byte source (USB CDC/UART StreamBuffer) with no NTRIP handshake — the embedded local-base entry point. Note: the stats scanner (rtcm3.scanFrames) drops a frame's preamble when a >=6-byte partial sits at a read boundary — telemetry-only, the ring relay is byte-transparent; documented in SourceFeeder.feed. Pre-existing behavior. Host build + all src tests (3 new: feeder tally / resync / local-source registration) + TCP integration green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The caster's portable code now cross-compiles for esp32p4 (riscv32-freestanding) with zero std.Thread/std.net/std.posix errors — the only remaining wall is the FreeRTOS/lwip headers, which exist only inside the ESP-IDF build. Host build + unit + TCP integration tests stay green (the posix backend is a 1:1 passthrough, behaviour-preserving). What was blocking: for `riscv32-freestanding` Zig has no OS, so std.Thread (pthread), std.net (sockaddr) and std.posix (IOV_MAX, ...) all collapse to `void`. io.zig already abstracted the socket Stream/Address surface; this extends the same backend-swap pattern to every other OS service the caster leans on. - os.zig / os_lwip.zig (new): Mutex / RwLock / ResetEvent / Thread / sleep. Posix aliases std.Thread.* (zero change); lwip backs them with FreeRTOS (semaphores / xTaskCreate / event groups). 11 files swapped std.Thread.* → os.*. - io.zig: Address is now backend-split (posix std.net wrapper vs a freestanding LwipAddress); new Listener/Accepted/shutdownHandle move the std.net.Server listen/accept + std.posix.shutdown behind the seam; new ms-precision socket-timeout + keepalive helpers replace the scattered std.posix.setsockopt calls (net/sockopt, server, fkp/*). - log.zig + embedded.zig: logging was the last hidden std dependency (std.log → std.debug stderr writer → pthread mutex + std.fs). Route it through os.consoleWrite and a std.Options.logFn override; file logging is posix-only. - embedded.zig (new): C-ABI caster_start() the firmware calls — wraps rtcm_sink_read() as a local Source feeder and runs the TCP listener. Its export fns are the link roots that force whole-tree analysis. - build.zig: `caster-lib` step builds the embedded static library (libntripcaster.a) for the ESP-IDF link. Remaining M2: the ESP-IDF component wiring in tab5-caster (provides the real FreeRTOS/lwip headers → true 0-error build + on-device). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Zig caster now cross-compiles cleanly for esp32p4 with the lwip
backend against the actual ESP-IDF FreeRTOS/lwip/newlib headers —
libntripcaster.a comes out as riscv32 / RVC / ilp32f, the same ABI
esp-idf uses, so it will link into the firmware. Host build + 237 tests
stay green (posix passthrough unchanged).
Feeding the real headers into @cImport surfaced the remaining
portability issues, now fixed:
- time: std.time.{milliTimestamp,timestamp} route through posix
clock_gettime (void on freestanding). Added os.milliTimestamp /
os.timestamp (posix: std.time; lwip: esp_timer_get_time, monotonic
since boot — the caster only takes differences). 22 call sites swapped.
- 64-bit atomics: rv32 has no 64-bit atomic RMW. Source.overrun_disconnects
was atomic(u64) → atomic(u32) (a disconnect counter; u32 is plenty).
- os_lwip.Thread: the completion semaphore used xSemaphoreCreateBinaryStatic,
whose macro doesn't translate cleanly (a NULL arg mistyped as [*c]u8) —
switched to the dynamic xSemaphoreCreateBinary().
- build.zig: caster-lib reads NTRIPCASTER_IDF_INCLUDES (the ESP-IDF
component build hands it FreeRTOS/lwip/newlib include dirs) and adds
them as system include paths for the backend @cImports.
Verified out-of-band with the real include set from the configured
esp32p4 build; the ESP-IDF component that automates this wiring is next.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
std.fmt's float formatter pulls in 128-bit division (__udivti3), which the riscv32 libgcc esp-idf links against does not provide — the firmware link failed with an undefined reference. Zig's compiler-rt has it, so bundle it into libntripcaster.a. With this the Zig caster links cleanly into the esp32p4 firmware (tab5-caster.bin builds). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Flashed to the Tab5; the Zig caster now boots, binds its lwIP listener on :2101, logs through the console, and runs stably alongside the monitor — no panics. Fixes found on hardware: - os_lwip.Thread: ESP-IDF's xTaskCreate takes the stack depth in BYTES, not StackType_t words — we were dividing by the word size and handing tasks a quarter of the intended stack. Pass cfg.stack_size straight through, and raise the default to 16 KiB: the caster was written for desktop (MiB stacks) and several hot paths put a 4 KiB buffer on the stack (runLocalSource's CHUNK_SIZE buf, handleConnection's header_buf), which overflows an 8 KiB FreeRTOS task. - embedded.zig: caster_start() did the config parse + ServerState.init on the caller's stack — the console REPL task's ~4 KiB — and overflowed it. Move all of it into a dedicated 16 KiB casterMain task; the export just spawns that and returns. - embedded.zig: read the caster feed from rtcm_caster_read (the firmware's tee) instead of rtcm_sink_read. A FreeRTOS StreamBuffer allows a single reader, so the monitor drain task and the caster feeder cannot share the primary sink; the firmware now tees the bytes into a second buffer. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds an export fn the firmware's `csource` command reads to confirm the caster's own feeder is draining the tee and the SourceFeeder is parsing RTCM3 — independently of the C-side monitor (which taps the primary sink). Returns bytes_in, rtcm_detected, client count, and up to 12 (msg_type, count) pairs from the live source. On-device this showed the full multi-GNSS MSM7 stream flowing into /MOSAIC (1077/1087/1097/1107/1117/1127/1137 + 1006/1033/1230, rtcm_detected=1, bytes_in climbing). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Introduces
src/io.zig, a thin backend-swappable I/O layer, as the first step toward running ntripcaster on M5Stack Tab5 (ESP32-P4 / RISC-V + ESP32-C6) as flashed, sellable firmware — while keeping the existing Linux/cloud binary building from the same source.Why
Every connection handler threaded
std.net.Stream/std.net.Addressend-to-end. Those depend on posix sockets/sockaddr and don't exist onriscv32-freestanding(ESP-IDF + lwIP + FreeRTOS). This wraps them so handlers only seeio.*, leaving a clean seam for a future lwIP backend.Changes
src/io.zig(new):io.Stream— surface is justhandle/read/writeAll/close. posix backend delegates 1:1 tostd.net.Stream(behavior unchanged).io.Address—initIp4/parseIp/parseIp4/getPort/format. Pure value type; lwip can swap to ansockaddr_in-based representation.io.tcpConnectToHost— outbound connect for the FKP upstream client.server,admin.server,ntrip.source,ntrip.client,fkp.vrs,fkp.upstream,fkp.runtime,net.sockopt,admin.stats.listen/acceptstay onstd.net.Server(backend-specific) and convert at the accept boundary viafromNet.-Dio-backend=posix|lwip(defaultposix).lwipis an@compileErrorseam markingio_lwip.zigTODOs (lwip_read/write/close,tcpConnectToHost).Design note
Target is 5–10 concurrent connections (a farm's auto-steer tractors + RTK drones), so the thread-per-connection model stays — no event-loop rewrite needed. Beyond that, deployments move to a cloud caster (same binary).
LWIP_MAX_SOCKETSwill need raising to ~24 on the embedded target (sockets = listeners + admin + upstream + bases + rovers).Verification
zig buildand-Doptimize=ReleaseSafe✅zig build test(unit + src) ✅zig build test-integration(TCP, run twice) ✅-Dio-backend=lwipcorrectly halts at the TODO@compileErrorseam ✅This is a pure refactor: the posix path is byte-for-byte the old behavior. No runtime logic changed.
Follow-up (needs hardware)
io_lwip.zig— lwIP socket implementationriscv32-freestandingstatic lib,heap_caps/PSRAM allocator, FreeRTOS tasks)sdkconfig(LWIP_MAX_SOCKETS, WiFi via C6/ESP-Hosted or Ethernet)🤖 Generated with Claude Code